Roc Analysis in the Evaluation of Intelligent Medical Systems

A large number of intelligent medical systems exist, but few are in routine clinical use. This is due, in part, to a lack of a robust objective method to quantify the performance of such systems. Potentially, ROC analysis could form a basis for a robust and objective evaluation of intelligent medical systems, but existing methods of ROC analysis require large sample sizes to be statistically valid. However, evaluation of intelligent medical systems often involve a small number of cases (because of time and cost of collecting ‘gold standards’) and so confidence bounds are required for ROC indices of performance. In this paper we present a new method for generating the probability density functions (pdfs) and confidence bounds for ROC points which is robust and accurate for any sample size. The method is generic and is particularly suited for evaluating the performance of systems where sample sizes are small. We illustrate the use of the method by applying it to assess the performance of two medical systems taken from the literature. The method has been implemented in C and in MATLAB.